Quality Control in Crowdsourcing: A Survey of Quality Attributes, Assessment Techniques and Assurance Actions

Crowdsourcing enables one to leverage on the intelligence and wisdom of potentially large groups of individuals toward solving problems. Common problems approached with crowdsourcing are labeling images, translating or transcribing text, providing opinions or ideas, and similar - all tasks that computers are not good at or where they may even fail altogether. The introduction of humans into computations and/or everyday work, however, also poses critical, novel challenges in terms of quality control, as the crowd is typically composed of people with unknown and very diverse abilities, skills, interests, personal objectives and technological resources. This survey studies quality in the context of crowdsourcing along several dimensions, so as to define and characterize it and to understand the current state of the art. Specifically, this survey derives a quality model for crowdsourcing tasks, identifies the methods and techniques that can be used to assess the attributes of the model, and the actions and strategies that help prevent and mitigate quality problems. An analysis of how these features are supported by the state of the art further identifies open issues and informs an outlook on hot future research directions.Comment: 40 pages main paper, 5 pages appendi

Supporting social innovation through visualisations of community interactions

Online communities that form through the introduction of sociotechnical platforms require significant effort to cultivate and sustain. Providing open, transparent information on community behaviour can motivate participation from community members themselves, while also providing platform administrators with detailed interaction dynamics. However, challenges arise in both understanding what information is conducive to engagement and sustainability, and then how best to represent this information to platform stakeholders. Towards a better understanding of these challenges, we present the design, implementation, and evaluation of a set of simple visualisations integrated into a Collective Awareness Platform for Social Innovation platform titled commonfare.net. We discuss the promise and challenge of bringing social innovation into the digital age, in terms of supporting sustained platform use and collective action, and how the introduction of community visualisations has been directed towards achieving this goal

Health-related Microbial Quality of Drinking Water in Kangavar, Western Iran

Evaluation of the microbial quality of drinking water can prevent the water-borne diseases outbreak that is one of the most important challenges in the world. Therefore, the aim of this study was to assess the seasonal variation of water-borne diseases prevalence associated with the microbial quality of drinking water and the comparison between rural and urban areas in Kangavar city, west of Iran. To accomplish this study, the results of the microbial quality of drinking water and cases of simple diarrhea, dysentery, typhoid and hepatitis A were received from all rural and urban health centers of the city during five years (2006-2010). To determine the relationship between diseases and microbial quality of water, Correlation instruction and Pearson correlation coefficient were used. The results showed that except hepatitis A, the incidence of all diseases in different areas (urban or rural) and seasons had significant relationship with microbial contamination of drinking water (P-value<0.05). The stronger relationship was observed in rural areas than in urban areas (except simple diarrhea) and in warm seasons than in cold seasons. With respect to the impact of the microbial quality of water on the incidence of dysentery and typhoid diseases, keeping up the quality of drinking water in places and times with high sensitivity (rural areas and warm seasons) should be considered strongly

An Analytic Approach to People Evaluation in Crowdsourcing Systems 2012-4

Worker selection is a significant and challenging issue in crowdsourcing systems. Such selection is usually based on an assessment of the reputation of the individual workers participating in such systems. However, assessing the credibility and adequacy of such calculated reputation is a real challenge. In this paper, we propose an analytic model which leverages the values of the tasks completed, the credibility of the evaluators of the results of the tasks and time of evaluation of the results of these tasks in order to calculate an accurate and credible reputation rank of participating workers and fairness rank for evaluators. The model has been implemented and experimentally validated

Detecting, Representing and Querying Collusion in Online Rating Systems 2012-3

Online rating systems are subject to malicious behaviors mainly by posting unfair rating scores. Users may try to individually or collaboratively promote or demote a product. Collaborating unfair rating \u27collusion\u27 is more damaging than individual unfair rating. Although collusion detection in general has been widely studied, identifying collusion groups in online rating systems is less studied and needs more investigation. In this paper, we study impact of collusion in online rating systems and asses their susceptibility to collusion attacks. The proposed model uses a frequent itemset mining algorithm to detect candidate collusion groups. Then, several indicators are used for identifying collusion groups and for estimating how damaging such colluding groups might be. Also, we propose an algorithm for finding possible collusive subgroup inside larger groups which are not identified as collusive. The model has been implemented and we present results of experimental evaluation of our methodology